Peak load shaving is a crucial concept in the field of electricity distribution, involving strategies to reduce the maximum demand during peak usage hours. This practice is vital for electrical distributors as it significantly lessens the burdens associated with high energy production and distribution costs during peak periods. By efficiently managing the peak load, the overall reliability and stability of the electrical grid are enhanced, benefiting both the distributors and consumers.
In addressing peak load challenges, demand response (DR) stands out as an effective approach. DR dynamically adjusts power consumption patterns in response to the electricity supply, making it a flexible solution for peak load management. This approach is particularly relevant in the context of commercial buildings, which are responsible for a substantial part of electricity consumption and carbon emissions. In the United States, for instance, commercial buildings constitute 35% of the total electricity consumption and emit 826 million metric tons of CO2, accounting for 16% of the nation's carbon dioxide emissions. A significant portion of this consumption, often estimated at around 30%, is wasted, underscoring the need for efficient energy use. The Heating, Ventilation, and Air Conditioning (HVAC) systems in these buildings are the primary consumers of this electricity. In the U.S., HVAC systems account for about 30.7% of the end-user electricity consumption, while in Australia, this figure reaches around 45%. Given their high flexibility and controllability, HVAC systems are key to implementing peak load shaving measures.
This study seeks to explore the potential of peak load shaving in a group of commercial buildings and assess the associated gains and losses when these buildings adopt DR strategies. It specifically focuses on two types of loads: controllable loads and HVAC loads. Controllable loads include office equipment, lighting, and a part of the energy demand from parked Electric Vehicles (EVs), which are typically higher during standard working hours (9 AM to 5 PM). These loads offer a viable opportunity to be reduced during peak periods. On the other hand, HVAC loads can be managed through techniques such as preheating or precooling buildings and slightly adjusting the preferred temperatures of the occupants.
The study employs two distinct sets of Reinforcement Learning (RL) algorithms to manage these loads. For HVAC load management, which requires a discrete action space, we use three algorithms: Discrete Soft Actor Critic (SACD), Discrete Proximal Policy Optimization (PPOD), and Dueling Double Deep Q Learning with Prioritized Experience Replay (D3QN). In contrast, for controllable load management, which operates in a continuous action space, we implement three continuous RL algorithms: Proximal Policy Optimization (PPO), Deep Deterministic Policy Gradient (DDPG), and Twin Delayed DDPG (TD3). The algorithms are evaluated using data from Montreal, focusing on the second week of January 2022.
The report is structured to first provide an in-depth analysis of the mathematical models for the two categories of loads in commercial buildings. Following this, a concise overview of the implemented algorithms' structure is presented. Finally, the results and their implications are discussed, offering insights into the effectiveness of the applied strategies in peak load shaving and the potential benefits and drawbacks for commercial buildings participating in DR programs.
This study aims to contribute to a better understanding of energy management in commercial buildings, highlighting the role of advanced algorithmic approaches in optimizing energy usage and assisting in peak load shaving. The findings could offer valuable guidelines for utility companies, building managers, and policymakers in devising more efficient and sustainable energy management strategies.
This study develops a dynamic mathematical model that enables consumers to adopt DR by adjusting their load and modifying the HVAC operation schedule. The consumer's load is divided into three categories. HVAC consumption is the first part and is scheduled such that the indoor temperature does not deviate significantly from occupants' preferred temperature. Moreover, this model considers winter days when HVAC tries to increase indoor temperature (in the case of a summer day, the term ( B{i,t}.h{i} ) must come with a negative sign). The second category is the controllable loads like lighting and water heating that can be performed at lower power. The third category is non-shiftable and non-controllable loads such as TV, refrigerators, and security and alarm service. The fourth category sets a constraint into the model, and this load must be covered. However, departing from the ideal consumption pattern for other types brings dissatisfaction. Therefore, this model minimizes the energy covering cost besides dissatisfaction arising from energy saving.
This model uses four indexes for time steps, peak periods, consumers and their appliances, respectively:
$$ \begin{align*} t \in& \{\mathcal{T}\} && \text{ episode } t \text{ in horizon } [1,2,3,...,T]\\ t \in& \{\mathcal{H}\} && \text{ episode } t \text{ in peak period, } (\mathcal{H} \subset \mathcal{T}) \end{align*} $$Below is the list of parameters:
$$ \begin{align*} C_{i} \doteq & \text{ Thermal capacitance of consumer $i$'s building (J/$^{\circ}$C) } \\ R_{i} \doteq & \text{ Thermal resistance of consumer $i$'s building ($^{\circ}$C/W)} \\ h_{i} \doteq & \text{ HVAC power rating of consumer $i$'s building (kW)} \\ T_{i,t}^{des} \doteq & \text{ Desired indoor temperature in consumer $i$'s building at time } t (^{\circ}C) \\ T_{t}^{out} \doteq & \text{ Outdoor temperature forecast at time } t (^{\circ}C) \\ e_{i,t}^{N} \doteq & \text{ Non-controllable load of consumer $i$ at time }t\\ e_{i,t}^{C} \doteq & \text{ Controllable load of consumer $i$ at time }t\\ P_{t} \doteq & \text{ Electricity price (\$/kWh) to buy from grid at time } t \\ \alpha_{i} \doteq & \text{ Consumer $i$'s discomfort weight factor for the temperature difference (\$/$^{\circ}C^2$))}\\ \beta_{i} \doteq & \text{ Consumer $i$'s discomfort weight factor for the reduced power of controllable load (\$/$^{\circ}C^2$)}\\ \delta \doteq & \text{ Importance (or equivalently cost) weight factor for the shaved peak load (Note that} \end{align*} $$The decision variable set below indicates the electricity flows and curtailed load in the consumers' problem; the list of the decision variables is as below:
$$ \begin{align*} B_{i,t} \doteq& \text{ HVAC status (on/off) in consumer $i$'s building at time } t\\ T_{i}^{in} \doteq & \text{ Indoor temperature ($^{\circ}C$) in consumer $i$'s building at time } t\\ E_{i,t}^{N} \doteq & \text{ Realized non-controllable load (kWh) of consumer $i$ at time }t\\ E_{i,t}^{C} \doteq & \text{ Realized controllable load (kWh) of consumer $i$ at time }t\\ E_{t}^{B} \doteq & \text{ Total energy consumption (kWh) in the reference model at time }t \end{align*} $$The reference model captures the behavior and energy consumption of the consumer when she does not adopt DR and consumes electricity maximizing her comfort regardless of its cost.
$$ \begin{align} \text{min} & \sum_{i,t} (T_{i,t}^{des} - T_{i,t}^{in})^2 \tag{1} \\ \text{subject to:} \\ T_{i,t}^{in} &= T_{i,t-1}^{in} + \left( \frac{T_{t}^{out}-T_{i,t-1}^{in}}{R_{i}}-B_{i,t}.h_{i}\right)\frac{\Delta t}{C_{i}} & \forall i,t \tag{2} \\ E_{i,t}^{N} &= e_{i,t}^{N} & \forall i,t \tag{3} \\ E_{i,t}^{C} &= e_{i,t}^{C} & \forall i,t \tag{4} \\ E_{t}^{B} &= \sum_{i,j} E_{i,t}^{N} + E_{i,t}^{C} + B_{i,t}.h_{i} & \forall t \tag{5} \end{align} $$In the reference model, objective function 1 keeps the indoor temperature as close as possible to the desired temperature. Note that as mentioned before, there is no cost term in this function. Equation 2 calculates indoor temperature in the next time step. Constraints 3 and 4 make sure all load types are covered at their preferred power. Finally, equation 5 calculates total energy consumed at time (t).
Once the reference model is solved and (E{t}^{B}) is calculated, the main model below takes (E{t}^{B}) as a parameter and uses it as a reference point to calculate the shaved peak load. The main model is as below:
$$ \begin{align} \begin{split} \min & \sum_{i,j,t} P_{t} (B_{i,t} \cdot h_{i} + E_{i,t}^{N} + E_{i,t}^{C}) \\ + & \sum_{i,j,t} \alpha_{i} (T_{i,t}^{\text{in}} - T_{i,t}^{\text{des}})^2 + \beta_{i} (E_{i,t}^{C} - e_{i,t}^{C})^2 \\ - \delta & \sum_{t \in \mathcal{H}} \left( E_{t}^{B} - \sum_{i,j} B_{i,t} \cdot h_{i} + E_{i,t}^{N} + E_{i,t}^{C} \right) \end{split} \end{align} \tag{6} $$Subject to:
$$ \begin{align} T_{i,t}^{in} &= T_{i,t-1}^{in} + \left( \frac{T_{t}^{out}-T_{i,t-1}^{in}}{R_{i}}-B_{i,t}.h_{i}\right)\frac{\Delta t}{C_{i}} & \forall i,t \tag{7}\\ E_{i,t}^{N} &= e_{i,t}^{N} & \forall i,t \tag{8}\\ E_{i,t}^{C} &\leq e_{i,t}^{C} & \forall i,t \tag{9} \end{align} $$Objective function 6 consists of three terms. The first term minimizes the electricity covering cost and the second term minimizes the dissatisfaction arisen by adopting DR and the difference between the desired and actual indoor temperature. Also, the third line maximizes the energy consumption reduction during the peak period. Constraint 7 calculates the indoor temperature in the next time step and constraints 8 and 9 make sure non-controllable load is covered and deducted load is less than or equal to the allowed bound.
The Reinforcement Learning (RL) algorithms and optimization models were run for the second week of January 2022 with a time step of 5 minutes. These RL algorithms share the same action and state spaces, but they differ in their application to controllable loads and HVAC loads.
For managing controllable loads, the state space of the RL algorithms includes four components:
The action vector for controllable loads comprises a continuous number ranging between zero and one, indicating the proportion of the load to be reduced.
For HVAC load management, the state space includes five elements:
The action space for HVAC load management includes a binary decision variable. If set to one, the HVAC system is turned ON, and if set to zero, it remains OFF.
This structure of state and action spaces allows the RL algorithms to make informed decisions about load management, balancing the need for energy savings with maintaining comfort and functionality in commercial buildings.
import plotly.graph_objects as go
import pickle
import numpy as np
import warnings
warnings.filterwarnings('ignore')
h = 10
with open(r'datasets/df.pkl', 'rb') as f:
df = pickle.load(f)
# Create an empty figure
fig = go.Figure()
# List of columns representing loads of different houses
load_columns = ['Other Load (kWh)', 'Reducible Load (kWh)', 'E_B (kWh)']
# Add a trace for each column
for col in load_columns:
fig.add_trace(go.Scatter(
x=df['Date/Time (LST)'],
y=df[col],
mode='lines',
stackgroup='one', # Define stack group
name=col
))
# Update layout
fig.update_layout(
title="Different Load Components Over Time",
xaxis_title="Date/Time (LST)",
yaxis_title="Load (kWh)",
showlegend=True
)
# Show the plot
fig.show(renderer='notebook')
#CTRL
# File names
files = ['res/rew_CTRL_DDPG.pkl', 'res/rew_CTRL_PPO.pkl', 'res/rew_CTRL_TD3.pkl']
algorithms = ['DDPG', 'PPO', 'TD3']
# Initialize a plot
fig = go.Figure()
# Loop through each file
for file, algorithm in zip(files, algorithms):
# Load data from the file
with open(file, 'rb') as f:
data = pickle.load(f)
# Extract rewards and negate them to make positive
rewards = [item.item() if isinstance(item, np.ndarray) else -item for item in data]
# Add a trace for each algorithm
fig.add_trace(go.Scatter(x=list(range(1, len(rewards) + 1)), y=rewards, mode='lines', name=algorithm))
# Update layout with y-axis range
fig.update_layout(title='Reward Over Episodes for DDPG, PPO, and TD3 Algorithms for Controllable Load',
xaxis_title='Episode',
yaxis_title='Reward',
yaxis=dict(range=[-500, 500]), # Set y-axis range
legend_title='Algorithms')
# Show the plot
fig.show()
#HVAC
# New file paths
files = ['res/rew_HVAC_D3QN.pkl', 'res/rew_HVAC_PPOD.pkl', 'res/rew_HVAC_SACD.pkl']
algorithms = ['D3QN', 'PPOD', 'SACD']
# Initialize a plot
fig = go.Figure()
# Check if files exist and process them
for file, algorithm in zip(files, algorithms):
# Load data from the file
with open(file, 'rb') as f:
data = pickle.load(f)
# Check if data is already in list format or wrapped in numpy arrays
if isinstance(data[0], np.ndarray):
# Extract rewards from numpy arrays and negate them to make positive
rewards = [item.item() for item in data]
else:
# Directly negate list values
rewards = [item for item in data]
# Add a trace for each algorithm
fig.add_trace(go.Scatter(x=list(range(1, len(rewards) + 1)), y=rewards, mode='lines', name=algorithm))
# Update layout with y-axis range
fig.update_layout(title='Reward Over Episodes for D3QN, PPOD, and SACD Algorithms for HVAC Management',
xaxis_title='Episode',
yaxis_title='Reward',
yaxis=dict(range=[-10000, 0]), # Set y-axis range
legend_title='Algorithms')
# Show the plot
fig.show()
import plotly.graph_objs as go
import pandas as pd
import pickle
env, alg = 'HVAC', 'SACD'
with open(f'res/tin_{env}_{alg}.pkl', 'rb') as f:
tin = pickle.load(f)
with open(r'datasets/df.pkl', 'rb') as f:
db = pickle.load(f)
df = pd.DataFrame({f"Week {i+1}": tin[i] for i in range(len(tin))})
T_out = db['Temp (°C)']
# Generate time steps for 5-minute intervals across a week
time_steps = pd.date_range(start="2023-01-08 00:00", periods=len(df), freq='5T')
# Define desired temperature
desired_temperature = 22
# Create frames for each week
frames = [go.Frame(data=[go.Scatter(x=time_steps, y=df[f"Week {i+1}"], mode='lines', name="Inside Temperature"),
go.Scatter(x=time_steps, y=T_out, mode='lines', name="Outside Temperature"),
go.Scatter(x=[time_steps[0], time_steps[-1]], y=[desired_temperature, desired_temperature], mode="lines", line=dict(color="red", dash='dash'), name="Desired Temperature")],
name=f"episode {i+1} * 2e3") for i in range(len(df.columns))]
# Set up the initial plot (first week)
fig = go.Figure(
data=[go.Scatter(x=time_steps, y=df["Week 1"], mode='lines', name="Inside Temperature"),
go.Scatter(x=time_steps, y=T_out, mode='lines', name="Outside Temperature"),
go.Scatter(x=[time_steps[0], time_steps[-1]], y=[desired_temperature, desired_temperature], mode="lines", line=dict(color="red", dash='dash'), name="Desired Temperature")],
layout=go.Layout(
title_text="Optimized Inside Temperature by SACD over Training Process",
yaxis=dict(range=[-40, 80]), # Setting the y-axis range
updatemenus=[dict(
type="buttons",
buttons=[dict(label="Play",
method="animate",
args=[None, {"frame": {"duration": 500, "redraw": True}, "fromcurrent": True}]),
dict(label="Pause",
method="animate",
args=[[None], {"frame": {"duration": 0, "redraw": False}, "mode": "immediate"}])
]
)]
),
frames=frames
)
# Add slider for episode selection
sliders = [dict(
steps=[dict(method='animate',
args=[[f"episode {i+1} * 2e3"],
{"frame": {"duration": 500, "redraw": True},
"mode": "immediate",
"transition": {"duration": 500}}],
label=f"episode {i+1} * 2e3")
for i in range(len(df.columns))],
transition={"duration": 300},
x=0,
y=0,
currentvalue={"visible": True, "prefix": "Episode: ", "xanchor": "center"},
len=1.0
)]
fig.update_layout(sliders=sliders)
fig.show()
import numpy as np
import pandas as pd
import plotly.graph_objs as go
import plotly.express as px
import numpy as np
import pickle
h = 10
def complete_week_dataset(act):
# Number of 5-minute intervals in a day
intervals_per_day = 24 * 12
# Create a full week vector filled with zeros
full_week = np.zeros(7 * intervals_per_day)
# Define the time slots in terms of 5-minute intervals
morning_slot = range(6*12, 9*12) # 6-9 AM
evening_slot = range(16*12, 20*12) # 4-8 PM
# Fill in the data from 'act' to the appropriate slots in 'full_week'
act_index = 0
for day in range(7): # Loop through each day of the week
for i in morning_slot:
full_week[day*intervals_per_day + i] = act[act_index]
act_index += 1
for i in evening_slot:
full_week[day*intervals_per_day + i] = act[act_index]
act_index += 1
return full_week
with open(r'datasets/df.pkl', 'rb') as f:
db = pickle.load(f)
with open(f'res/act_CTRL_PPO.pkl', 'rb') as f:
act = pickle.load(f)
with open(f'res/HIST_HVAC_SACD.pkl', 'rb') as f:
hist = pickle.load(f)
db['Reduced Load (kWh)'] = (1 - complete_week_dataset(act[-1])) * db['Reducible Load (kWh)']
# db['Reduced Load'][db['Reduced Load'] < 0] = 0
db['HVAC Load (kWh)'] = np.array(hist) * h
db['Optimized Load (kWh)'] = db['Reduced Load (kWh)'] + db['HVAC Load (kWh)'] + db['Other Load (kWh)']
db['Base Load (kWh)'] = db['E_B (kWh)'] + db['Reducible Load (kWh)'] + db['Other Load (kWh)']
db['Load Difference (kWh)'] = db['Optimized Load (kWh)'] - db['Base Load (kWh)']
db['Date/Time (LST)'] = pd.to_datetime(db['Date/Time (LST)'])
# Set the 'Date/Time (LST)' as the index
db.set_index('Date/Time (LST)', inplace=True)
# Resample the data to hourly intervals
hourly_data = db.resample('30T').sum()
# Create the plot
fig = px.line(hourly_data, x=hourly_data.index, y=['Optimized Load (kWh)', 'Base Load (kWh)'],
title='Optimized Load vs. Base Load (kWh) (green: 6-9am, blue: 4-8pm)',
labels={'value': 'Load (kWh)', 'Date/Time (LST)': 'Time'})
# Get unique dates
unique_dates = hourly_data.index.normalize().unique()
# Add shaded areas for specific time ranges for each day
for date in unique_dates:
# Format date to string for use in x0 and x1
date_str = date.strftime('%Y-%m-%d')
# Highlight 6-9 AM with green color
fig.add_vrect(x0=f"{date_str} 06:00:00", x1=f"{date_str} 09:00:00", fillcolor="green", opacity=0.2, line_width=0)
# Highlight 4-8 PM with blue color
fig.add_vrect(x0=f"{date_str} 16:00:00", x1=f"{date_str} 20:00:00", fillcolor="blue", opacity=0.2, line_width=0)
# Show the plot
fig.show()
# Calculating results
db['T_in (RL)'] = tin[-1]
optimized_cost = (db['Reduced Load (kWh)'].sum() + db['HVAC Load (kWh)'].sum() + db['Other Load (kWh)'].sum()) * 0.1
base_cost = (db['Reducible Load (kWh)'].sum() + db['E_B (kWh)'].sum() + db['Other Load (kWh)'].sum()) * 0.1
red_load_ration = db['Reduced Load (kWh)'].sum() / db['Reducible Load (kWh)'].sum()
temp_diff= np.sum(db['T_in (RL)'] - 22)/2016
# Convert the 'Date/Time (LST)' column to datetime if it's not already
db['Date/Time (LST)'] = pd.to_datetime(db.index)
# Filter for 6-9am and 4-8pm
morning_filter = db['Date/Time (LST)'].dt.time.between(pd.to_datetime('06:00:00').time(), pd.to_datetime('09:00:00').time())
evening_filter = db['Date/Time (LST)'].dt.time.between(pd.to_datetime('16:00:00').time(), pd.to_datetime('20:00:00').time())
# Apply filters
morning_data = db[morning_filter]
evening_data = db[evening_filter]
# Calculate sum of differences for each time period
morning_sum = (morning_data['Base Load (kWh)'] - morning_data['Optimized Load (kWh)']).sum()
evening_sum = (evening_data['Base Load (kWh)'] - evening_data['Optimized Load (kWh)']).sum()
# Total sum
total_sum = morning_sum + evening_sum
base_max = np.max([morning_data['Base Load (kWh)'].max(), evening_data['Base Load (kWh)'].max()])
opt_max = np.max([morning_data['Optimized Load (kWh)'].max(), evening_data['Optimized Load (kWh)'].max()])
print(f'cost reduction: %{np.round(100*(1-optimized_cost/base_cost),2)}')
print(f'%{np.round(100*(1-red_load_ration),2)} of reducible load is deducted')
print(f'on average temperature is {np.round(-temp_diff, 2)} degree (°C) below preferred temperature.')
print(f'total reduced load during peak hours: {np.round(total_sum, 2)} kWh')
print(f"ratio of reduced load to total building's load: %{np.round(100*total_sum/db['Base Load (kWh)'].sum(),2)}")
print(f'{np.round(base_max - opt_max, 2)} (kWh) reduction in peak load during peak hours')
print(f'reduction percentage in peak load during peak hours: %{np.round(100*(base_max - opt_max)/base_max, 2)}')
# Creating a plot for the results that are in percentage format
# Data for the plot
percentage_results = {
"Metrics": ["Cost Reduction", "Reducible Load Deducted", "Reduction in Peak Hour Consumption", "Reduction in Peak Load"],
"Values": [np.round(100*(1-optimized_cost/base_cost),2), np.round(100*(1-red_load_ration),2), np.round(100*total_sum/db['Base Load (kWh)'].sum(),2), np.round(100*(base_max - opt_max)/base_max, 2)] # Percentages
}
# Convert to DataFrame
df_percentage = pd.DataFrame(percentage_results)
# Define different colors for each bar
colors = ['red', 'green', 'blue', 'orange']
# Create a figure
fig = go.Figure()
# Add a bar for each metric with a different color
for i, color in enumerate(colors):
fig.add_trace(go.Bar(
x=[df_percentage["Metrics"][i]],
y=[df_percentage["Values"][i]],
marker_color=color,
name=df_percentage["Metrics"][i] # This will add a legend for each bar
))
# Update layout
fig.update_layout(
title='Percentage-Based Results with Different Colors',
xaxis_title='Metric',
yaxis_title='Percentage (%)',
yaxis=dict(range=[0, max(df_percentage["Values"]) + 5]) # Adding some space above the highest bar
)
# Display the figure
fig.show()
average_temp_below_preferred = np.round(-temp_diff, 2)
fig = go.Figure(
go.Indicator(
mode="gauge+number",
value=average_temp_below_preferred,
title={'text': "Average Temperature Below Preferred (°C)"},
gauge={'axis': {'range': [None, 5]}}
)
)
# Adjust the size of the figure
fig.update_layout(width=600, height=300) # Set your desired width and height
fig.show()
cost reduction: %4.43 %6.32 of reducible load is deducted on average temperature is 0.75 degree (°C) below preferred temperature. total reduced load during peak hours: 2865.42 kWh ratio of reduced load to total building's load: %8.37 2.3 (kWh) reduction in peak load during peak hours reduction percentage in peak load during peak hours: %4.76
These resutlsbelong to a building with 40-50 occupant. More buildings with different load pattern and HVAC charachteristics will be considered. It will firstly make the results more robust against variations in parameters of the problem and secondly, it will answer how much will a cluster of building can provide peak load reduction and how it scales, and to what extent their load and behaviour overlap. and then a paper will be drafted.